The present disclosure relates to testing coverage in general, and to increasing functional coverage of a testing phase, in particular.
Computerized devices control almost every aspect of our life—from writing documents to controlling traffic lights. However, computerized devices are bug-prone, and thus require a testing phase in which the bugs should be discovered. The testing phase is considered one of the most difficult tasks in designing a computerized device. The cost of a bug may be enormous, as its consequences may be disastrous. For example, a bug may cause the injury of a person relying on the designated behavior of the computerized device. Additionally, a bug in hardware or firmware may be expensive to fix, as patching it requires call-back of the computerized device. Hence, many developers of computerized devices invest a large portion, such as 70%, of the development cycle to discover erroneous behaviors of the computerized device.
During the testing phase, a sample of all possible behaviors of the computerized device is inspected. Modern testing techniques focus on testing different behaviors, by measuring coverage of the executed tests. Functional coverage is one such method. It is designed to measure amount, portion or a similar metric of tests that examined predetermined functional behaviors. Once functional coverage is measured, quality assurance (QA) personnel may design additional tests to examine untested behaviors.
Another testing technique is combinatorial test design, in which a model of possible behaviors of the computerized device is formulated and is used to generate tests that are configured to cover a subset of all possible behaviors. For example, a model may define possible configurations of a system comprising an operating system (OS), a web-browser, a mail server and a central processing unit (CPU). The OS may be, for example, Linux, Microsoft Windows XP or Microsoft Windows Vista. The web-browser may be, for example, Mozilla Firefox, Google Chrome, Microsoft Internet Explorer or Opera web browser. In a similar manner, there may be a predetermined number of mail servers and CPUs. A test of each possible combination of the above mentioned components may be created, however, in order to reduce the number of tests to be executed, a combinatorial test design may be applied. A model may define or characterize subsets of the entire test-space to cover. The model may additionally define a subset of the entire test-space that is invalid. For example, a known incompatibility between Linux OS with the Microsoft Internet Explorer web browser. Thus enabling generation of tests that are configured to cover a subset of all possible behaviors of the system. The model may be defined to cover a small subset of all possible behaviors, such that the small subset has predetermined properties that are deemed favorable by, for example, QA personnel.